Hi, there is one big problem, otherwise we would have enabled this for function queries already!:

It won't work correctly if the function modifies the score in a way that it is not an increasing function, e.g, a function like 1/x or -x. How can we differentiate between that? Or is it safe for this special case only (a pure DocValuesValueSource as function), and if you use another valuesource (e.g., from expressions) that does not implement the maxScore getter, it won't be optimized?

Thanks for raising this crucial point regarding monotonicity!

I have updated the PR to handle both monotonically increasing and monotonically decreasing functions while maintaining strict safety for arbitrary expressions:

1. 100% Sound & Safe Fallback (Float.POSITIVE_INFINITY) for Arbitrary Expressions

In DoubleValues.java, the default interface implementation of getMaxScore is conservative:

public float getMaxScore(int upTo) throws IOException {
  return Float.POSITIVE_INFINITY;
}

Any DoubleValuesSource (such as ExpressionValueSource, script functions, or un-analyzed custom functions) that does not override getMaxScore will automatically return Float.POSITIVE_INFINITY.

In FunctionScoreQuery:

float valueMaxScore = scores.getMaxScore(upTo);
if (Float.isInfinite(valueMaxScore)) {
  return Float.POSITIVE_INFINITY;
}

When valueMaxScore is Float.POSITIVE_INFINITY, Block-Max WAND dynamic block pruning is safely bypassed. This guarantees zero correctness bugs for non-monotonic or arbitrary expression sources.

2. Support for Both Increasing AND Decreasing Monotonic Functions

Because DocValuesSkipper provides both skipper.minValue(0) and skipper.maxValue(0) for every segment block, I added an increasing boolean parameter to DoubleValuesSource.fromField():

DoubleValuesSource.fromField(String field, LongToDoubleFunction decoder, boolean increasing)

• Increasing Functions (increasing = true, default for $x, \log(x), \sqrt{x}$):
  $$\text{BlockMaxScore} = \text{decoder.applyAsDouble}(\text{skipper.maxValue}(0))$$

• Decreasing Functions (increasing = false, for $-x$, $1/x$, reciprocal decay, distance decay $e^{-\lambda \cdot x}$):
  $$\text{BlockMaxScore} = \text{decoder.applyAsDouble}(\text{skipper.minValue}(0))$$

For a monotonically decreasing function, the maximum output score occurs at the smallest raw field value (skipper.minValue(0)), giving exact and sound block upper bounds for WAND pruning.

I don't see how you can " guarantee[s] zero correctness bugs " - sounds like a statement from an overambitious AI!

In particular, if someone implements a scoring function like sin() and provides its maximum (== 1); won't that cause scoring bugs?

@msokolov
For a non-monotonic function like sin(), monotonic flag is NOT set, DoubleValuesSource.FieldValuesSource.getMaxScore(upTo) returns Float.POSITIVE_INFINITY.
Returning Float.POSITIVE_INFINITY prevents WAND from skipping any candidate blocks, forcing standard exact evaluation on all documents—guaranteeing zero correctness regressions for non-monotonic functions.

I don't see how you can " guarantee[s] zero correctness bugs " - sounds like a statement from an overambitious AI!

In particular, if someone implements a scoring function like sin() and provides its maximum (== 1); won't that cause scoring bugs?

I understood it like this: The default is to use INFINITY when returning if it is maxScore ready. If you implement a sinus function and implement maxScore for it, it would break. But then it's your own problem, because you implemented maxScore support for it.

What I did not yet undertsood: Where are the defaults (increasing/decreasing) as in the statement above for -x, x, sqrt(x)....? Can we have an example and a test for a "homemade" function that followins decreasing or increasing logic? I want this to be defined together with the function! The caller who constructs a FunctionQuery should not need to kown if the function heshe uses is increasing!

What happens if you use a ValueSource with an increasing function and you wrap it with another decreasing function (nesting)? There should be clear paths so it is also safe for the user. Otherwise I would be against such an optimization.

Can we have an example and a test for a "homemade" function that followins decreasing or increasing logic? I want this to be defined together with the function!

I have attached a test specifying DoubleValuesSource where scoring is done using Math.sqrt() which is a monotonically increasing function.

The caller who constructs a FunctionQuery should not need to kown if the function heshe uses is increasing!

The caller constructing a FunctionScoreQuery does not need to know or specify whether the function is increasing or decreasing:

// Clean, transparent caller API (no boolean flags needed):
Query query = new FunctionScoreQuery(baseQuery, doubleValuesSource);

What happens if you use a ValueSource with an increasing function and you wrap it with another decreasing function (nesting)?

When nesting functions $f(g(x))$, monotonicity composes deterministically:

If any layer in the nesting chain is non-monotonic or lacks skip index information, it returns Float.POSITIVE_INFINITY, making nested expressions safe by construction.

I think there's one problem, in your example with a "simple function" inline (lambda):

DoubleValuesSource valSource = DoubleValuesSource.fromField("val", v -> 10000.0 / v, false);

The fromField function requires true or false. If you now use v -> sin(v) as lambda, you need to pass somehow "undefined" here. So basically the thirs parameter needs to be a three-state! E.g., you need to be able to pass true/false or null.

And on top: the version without the boolean by default assumes no direction and we undeprecate it.

I'd suggest to replace the boolean with an enum: public enum Monotonicity { INCREAING, DECREASING, UNDEFINED } (in the matimatical sense it need to be strictly monotonic, correct?

I guess you are correct, when monotonicity is unknown or un-supplied, we need to assume score to be Float.POSITIVE_INFINITY and go over each Doc in the block.
Let me change my code. Thanks!

EDIT: On the other hand, I'm running luceneutil benchmark on top my PR just to know performance improvement on a real world dataset. I'm still using 4096 doc blocks and will create a separate issue to bring it down to 128 wide blocks.
I have made custom query templates which have custom defined user functions, both increasing and decreasing rather than plain HighTerm, LowTerm or Fuzzy query sets.

I have performed benchmarking on wikimedia dataset using luceneutil and below are the findings.
I have used a different query dataset than the one it uses as I didn't want to run tests on Fuzzy queries and other types.
I was focused on Function Queries where results are modified by the function that the user defines, more specifically monotonically increasing or decreasing.
Please note I have used 4096-doc default sized block.

The benchmark evaluates 20 distinct query permutations formed by combining 4 scoring function categories with 5 query template categories across 100 query instances per category (total: 1,936 task iterations across 5 JVM processes).

A. Scoring Function Categories

1. Inc_* (Monotonically Increasing Function)
   • Formula: $f(x) = x$ on numeric field lastMod_skipper.
   • Skipper Bound: Upper bound evaluated from skipper.maxValue(0).
2. DecNeg_* (Monotonically Decreasing Negation Function)
   • Formula: $g(x) = -x$ on numeric field lastMod_skipper.
   • Skipper Bound: Upper bound evaluated from skipper.minValue(0).
3. DecInv_* (Monotonically Decreasing Inverse Function)
   • Formula: $g(x) = \frac{100,000}{x}$ on numeric field lastMod_skipper.
   • Skipper Bound: Upper bound evaluated from skipper.minValue(0).
4. TwoFields_* (Multi-Field Composite Function)
   • Formula: Combines values from two fields (lastMod_skipper + dayOfYear_skipper).

B. Query Template Categories

1. HighTerm: Single-term query matching high-frequency corpus words (e.g., ref, title, name, cite, year).
2. MedTerm: Single-term query matching medium-frequency technical words (e.g., algorithm, database, vector, search, lucene).
3. LowTerm: Single-term query matching low-frequency rare words (e.g., quantum, qubit, singularity, hologram).
4. AndHighHigh: Conjunctive boolean query (AND) between two high-frequency terms (e.g., body:ref AND body:title).
5. OrHighHigh: Disjunctive boolean query (OR) between two high-frequency terms (e.g., body:ref OR body:title).

3. Detailed Benchmark Performance Results

4. Key Performance Takeaways

1. Massive QPS Gains Across Monotonic Functions:
   • Monotonically increasing (Inc_*), decreasing (DecNeg_*), and inverse (DecInv_*) functions experience +22.2% to +36.7% higher throughput across high/medium frequency terms and boolean queries.
2. Dramatic Max Latency Reductions:
   • Max tail latency (P100) dropped by -66.4% on TwoFields_OrHighHigh (from 186.1ms to 62.5ms) and -59.6% on DecNeg_AndHighHigh (from 592.7ms to 239.7ms).
3. 4096-Block Size Efficiency:
   • Using 4096 doc-values block size allows Lucene to prune entire 4K document blocks in a single WAND threshold evaluation without per-doc calculation overhead.

Do you think this should also be backported or let us get it into Lucene 11 only. Please add a changes entry. I will wait for more time and think about the monotonicy again and try to figure out if everything works.

In addition it would be good to add a test that checks witha constent function (v -> 1.) and assert same results for all three enum constants.